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We have searched for the flavor-changing neutral current decay b — ► X a /i fi~ in pp collisions at 
\fs =1.8 TeV with the D0 detector at Fermilab. We determine the 90% confidence level limit for the 
branching fraction to be B(b — > X s fi + fi~) < 3.2xlCF 4 . We argue that this limit is more stringent 
than the best published limit on this decay rate. 



2 



In the Standard Model of electroweak interactions 
(SM) the decay processes b — > X s [i + (where X s stands 
for a hadronic state containing a strange quark) and 
B° — > p, + fi~ are forbidden at tree level and are possible 
only through loop diagrams. The largest contributions 
to the branching fraction for these processes come from 
diagrams involving the top quark and therefore the pre- 
dicted branching fractions depend on the top quark mass. 
The "rare" flavor-changing neutral current (FCNC) de- 
cays are expected to be more frequent for b quarks than 
for strange quarks. The b — > c transition is suppressed by 
\Vbc\ — O(10 _1 ) while loop corrections are large due to 
Vfb| ~ 1 and the large top quark mass, rrit- At m t = 170 
GeV/c 2 , the SM expected branching fraction (TJd] for the 
semi-inclusive decay b — > X s fi + n~ is 6xlCP 6 . 

Extensions to the minimal Standard Model which al- 
low new particles contributing to the higher order correc- 
tions, such as fourth generation quarks, charged Higgs 
bosons or supersymmetric particles, provide additional 
possible sources of FCNC. Precision measurements of 
rare b decay rates thus extend the new physics discovery 
potential beyond direct searches. We describe a search 
for such decays in the data collected with the D0 detec- 
tor H during the 1994-1995 Fermilab Tevatron run. The 
data correspond to a total integrated luminosity of C = 
50.0 ± 2.7 pb- 1 . 

Dimuon events are selected by requiring two muons in 
the central muon system, at both hardware and software 
trigger levels |1. We select events containing an oppo- 
sitely charged muon pair with the invariant mass M M/i < 
7 GeV/c 2 , transverse momentum p^ > 5 GeV/c, and 
pseudorapidity |?7 MM < 0.6. The muons are required to 
have a transverse momentum pt^ > 3.5 GeV/c and pseu- 
dorapidity 1 77^ | < 1.0. 

Both muon trajectories are required to be consistent 
with the reconstructed vertex position and to have a 
matching track in the central detector. There must be 
appropriate energy deposition along the muons' path in 
the calorimeter. The total number of events satisfying 
the above criteria is 1564. 

The dimuon mass spectrum of these events is shown 
in Fig. ^. In addition to the J/ip resonance, the major 
known sources JH) of dimuons for < 7 GeV/c 2 are: 
(1) bb and cc events with both heavy quarks decaying 
semileptonically or with a sequential semileptonic decay 
b —> e+ fx, c — > s + /i; (2) the case where one muon comes 
from a b or c decay and the other from the decay of a 7r or 
K meson, and (3) virtual photon decays (the Drell-Yan 
process). 

The curve in Fig. [I](a) shows the results of a maximum 
likelihood fit of a sum of the J/ip signal and processes 
(l)-(3) to the dimuon mass spectrum in the range 1 < 
M^it < 7 GeV/c 2 . By J/ip signal we mean the J/ip plus 
an admixture of tp'. We use the ratio N(ip')/N(J/ip) = 
(6±2)%, based on the recent results from the CDF col- 
laboration MM , corrected for the mass dependence of the 
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FIG. 1. (a) Dimuon invariant mass spectrum. The solid 
line is the maximum likelihood fit of the known physics pro- 
cesses to the data (see text); (b) the data points after subtrac- 
tion of the fitted values, divided by the fitted values. Only 
statistical uncertainties on the data points are shown. The 
dashed line corresponds to the 90% confidence level upper 
limit for the decay b — > X s fi + obtained from the fit. The 
arrows indicate the search window for the decay b — > X s fj, + fi~ . 

kinematic acceptance for dimuons at D0. The ip' mass is 
assumed to be higher than the J/ip mass by 0.59 GeV/c 2 
and its width is assumed to be 20% larger than the 
J/?/ 1 width. The normalized M w distributions for pro- 
cesses (l)-(3) were obtained by fitting the corresponding 
Monte Carlo (MC) simulated spectra. We use a dimuon 
mass resolution function parametrized by a superposition 
of two Gaussians, with the shape determined by fitting 
MC simulated events of the decay J/ip — » p + fj,~ and the 
mass scale allowed to vary. For a five parameter fit to 
60 bins in Fig. |l|(a), we obtain x 2 (55)=56. The overall 
quality of the fit gives us confidence in the simulation of 
the dimuon mass resolution and the contributing physics 
processes. 

The search for the decay b — » is performed in 

the mass window 3.9 < M p/J < 4.9 GeV/c 2 . This mass 
range is above most of the known sources of dimuons and 
hence constitutes the region of maximum sensitivity to 
the b — > X s fj, + p,~ decay. Although we do not identify 
the strange particle among the hadrons originating from 
the b decay, we assume that the decay b — -» X s fi + /i~ 
dominates over the corresponding CKM-suppressed de- 
cay b — > Xdfj, + fj,~. 

We observe 56 events in our search window, where 
68±2(stat.)±4(syst.) are expected from the fit. We thus 
find no evidence for an excess of events to be attributed 
to the decay b — > X s /z + /i~. To estimate the systematic 
error, we have performed alternative fits, changing within 
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their uncertainties the width and skewness of the dimuon 
mass resolution function and mass scale, as well as the 
mix of backgrounds. MC studies of 10,000 simulated ex- 
periments, with the background composition as obtained 
in the fit to the data, indicate that the probability of 
obtaining 56 events in a given experiment where 68±5 
events are expected is 12%. The contribution to the \ 2 
from the 10 bins within the search window is 7.3. 

We follow two independent ways of relating the ob- 
served number of events in the search window to the 
expected b quark yield. First, we use the absolute nor- 
malization to the inclusive b quark production cross sec- 
tion. The result depends on the assumed b cross section, 
on the b production model and on the estimates for the 
trigger and offline reconstruction efficiencies. A similar 
method has been employed previously by the UA1 Col- 
laboration j^] . In the alternative approach, we normalize 
to the observed J/ip signal, where the uncertainties in 
the variables used in the denominator of Equation (1) 
largely cancel. The result of the latter method depends 
on the knowledge of the fraction of J/ip events that orig- 
inate from b decays, and on the branching fraction for 
the decay sequence b — > X s J/tp, J/ip — > pp. 

In the first approach, the branching fraction for the 
decay b —> X s p + p~ is given by 



B(b^X s p + p~) 



N 



2 ■ a(b) C-e 



(1) 



where N is the number of events due to this decay, a(b) 
denotes the inclusive b quark production cross section 
for pr(b) > 6 GeV/c and rapidity \y(b)\ < 1, and e is the 
combined kinematic acceptance and trigger and offline 
reconstruction efficiency. 

The inclusive b quark production cross section for pr(b) 
> 6 GeV/c and \y{b)\ < 1 is a(b) = 7.2 ± 1.8 ph. This 
estimate results from a fit to a compilation of D0 mea- 
surements Q of the integrated, inclusive b production 
cross section at pr(b) > p^ m . 

The calculation of e proceeds by multiplying the the- 
oretical mass spectrum j2j, normalized to unity, by the 
mass-dependent detection efficiency, convoluting the de- 
tector resolution, and integrating the resulting distribu- 
tion over the search window. The simulated mass spec- 
trum including the detector response is compared to the 
input distribution from Ref. § in Fig. ||. 

The mass-dependent dimuon detection efficiency is de- 
termined from events with b quarks generated with the 
ISAJET program jll| in the lowest order QCD approxi- 
mation. Quarks that satisfy the above kinematic require- 
ments are fragmented according to the Peterson fragmen- 
tation model |l2|]. We adjust the value of the fragmen- 
tation parameter t\, to obtain the dimuon transverse mo- 
mentum spectrum that matches the pt spectrum of J / ip 
coming from b quark decay, measured by the CDF Col- 
laboration 101. 
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FIG. 2. (a) The calculated differential branching fraction 
for the decay b — * X a fj, + /i~, from Ref. [2] as a function of 
M^y, (multiplied by 0.2 GeV/c 2 ); (b) the same differential 
branching fraction modified by the response of the D0 de- 
tector (multiplied by 0.1 GeV/c 2 ). The arrows indicate the 
search window used in this analysis. 



To expedite the simulation procedure, preselection cuts 
(called JC) of p^. > 3 GeV/c and < 1.0 are applied 
to both muons. The acceptance A{M I1)1 ) for this prese- 
lection increases with M MM . It is determined by studying 
MC samples of the decay b — > X s fi + fi~ generated at 
various values of M^. At the parton level, the decay 
b — » X s [i + fi~ has the same final state as the b decay to 
J/ip, as illustrated in the diagrams below: 




Therefore, we use "J/'0 + X s ", simulated according 
to Ref. pd] |, as a model for the final state of the decay 
b — > X s fi + pT by substituting various discrete values for 
the J/ip mass. At = 4.1 GeV/c 2 we remove multi- 
body channels, keeping the channels J/ipK and J/ipK* 
with the relative rate 1:2. We find the resulting accep- 
tance insensitive to the number of channels included and 
their relative rates. We also calculate the acceptance for 
the exclusive B meson decay to a muon pair. The results 
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arc fitted with the form A(M MM ) = 2.9xl(T 3 xM wl 2 - 64 . 
The uncertainty on A(4.1), taken from the difference be- 
tween its values at efc=0.02 and at e;,=0.006, is 20%. The 
uncertainty of the ratio of .4(4.1) to A(3.1) is 5%. 

Events that satisfy the muon pseudorapidity and mo- 
mentum cuts (/C) are passed through a detector simula- 
tion, trigger simulation, and offline reconstruction pro- 
grams. We find the effect of the trigger and reconstruc- 
tion efficiency for those events to be independent of M M/1 . 
The trigger efficiency, corrected for effects not included 
in the simulation, is 0.052 ± 0.005. The efficiency of the 
offline selection cuts, 0.19 ± 0.03, has been obtained by 
comparing the total number of J/ip events passing the 
dimuon trigger, 3310 ± 500, to the number of triggered 
J /ip events that satisfy our offline selection cuts, 633 ± 
45. The product of the trigger and offline selection effi- 
ciencies is £dct =(1-0 ± 0.2) xl0~ 2 . 

The integral of the spectrum in Fig. ||(b) over the 
search window is e = (7.0 ± 2.0)xl0~ 5 . The acceptance 
of the kinematic selection alone, including the selection 
(K.) and the mass cut, is (7.0 ± 1.4) xl0~ 3 . 

The J/ip signal allows an alternative normalization, 
to the J/ip yield due to the b quark decay, N b ^j^. 
The ratio of the kinematic acceptance A(M tltl ) integrated 
over the search window to 4(3.1) is a — 0.123 ± 0.006. 
From the total number of J/ip events in our dimuon sam- 
ple, 633 ± 45, and the fraction of J/ip events originat- 
ing from b decay, /& = 0.31 ± 0.03 the number 
of J/ip events coming from b decays is N b ^j^— 196 ± 
23. With the branching fraction for the decay sequence 
b -> X s J/ip, J/ip -► wi, B{b -> J/ip -> fifi) = (7.0±0.6) 
x 10~ 4 fi, we have: 
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FIG. 3. 
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N b ^,, N ,/B(b -» J/ip -» mj.) 
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The respective results for the denominators in Eqs. 
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(a) The invariant mass distribution for isolated 
dimuons. The solid line is the maximum likelihood fit of the 
known physics processes to the data (see text); (b) the data 
points after fit subtraction divided by the fitted values. The 
dashed line corresponds to the 90% confidence level upper 
limit for the decay B° — > /i + fjT obtained from the fit. The 
arrows indicate the search window for the decay B° — > (i + fx~ . 

efficiency of 0.011 we obtain M| euAi ~ 5.8 x 10 -4 - lower 
by a factor w 20. Differences between the theoretical 
dimuon mass distributions for the decay b — > X s fi + ji~ , or 
in the versions of the ISAJET program that were used here 
and in Ref . j^] , cannot account for such a large disparity 
in the results. Using our estimates of their efficiency we 
obtain psIxIO -3 as the upper limit on B(b — > X s fi + fi~) 
from their experiment. 

For the exclusive decay B — > (an unseparated 

mixture of Bd and B s decays) we define the search win- 



(1) and (2), (5.0± 1.9) xl0 4 and (3.4 ± 0.5) xlO 4 , are dow as 4 g < M ^ < 5 8 GeV / c 2 , resulting in the max- 



consistent. Using the latter by virtue of its smaller sys- 
tematic error, we obtain a 90% confidence level limit of 
B(b -> X„fi + /j,-) < 3.2 x 10~ 4 . To derive the limit, we 
apply a Bayesian approach in which the observed number 
of events is compared to the number of background events 
in the region of interest. We assume Poisson statistics for 
the signal and background and account for uncertainty 
in the background and in the estimates of the total cross 
section and of the dimuon detection efficiency. We have 
found the results to be stable with respect to the choice of 
the search window by varying the lower and upper limits 
of the window within ± 50 MeV/c 2 and ± 200 MeV/c 2 , 
repsectively. 

The best published limit for this decay ( 5xl0 -5 ) was 
set by the UA1 Collaboration ||. We have attempted 
to reproduce the UA1 limit and to make the cross-check 
between their quoted efficiency and their J / ip signal Q . 
We have failed to reconcile the two. Instead of the quoted 



imum sensitivity to the signal. The acceptance for this 
mass window is 0.60 ± 0.03. In this process, the two 
muons are expected to carry a large fraction of the en- 
ergy in a cone around the direction of the parent b quark. 
To reduce background, we select events whose energy de- 
position in the calorimeter in a cone around each muon of 
radius A1Z= 0.4 in the pseudorapidity - azimuthal angle 
space is less than 8 GeV. The acceptance of the isolation 
requirement is 0.80 ± 0.03. 

The mass spectrum for isolated dimuons is shown in 
Fig. ||(a). We find 15 events in the search region. From 
a fit to the sum of the J/ip signal and processes (l)-(3), 
the background in the search window is estimated to be 
15 ± 2 events. 

The B° production cross section at pr{B) > 6 GeV 
and |y(S)| < 1, measured by the CDF collaboration Jl6| , 
is 2.39 ± 0.54 fib. The product of the acceptance for 
kinematic and geometric restrictions on the two muons 
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coming from the decay of a B° meson, the trigger and of- 
fline reconstruction efficiency, and the mass and isolation 
cuts is e = (1.4 ± 0.35) xlCT 3 . We obtain a 90% confi- 
dence level limit of B(B° -> (J. + fJ,~) < 4.0 x 10~ 5 . The 
best published limit for this decay is 1.6xl0 -6 , set by 
the CDF Collaboration in Rcf. (l6). The SM prediction 
is 1.5 xl0~ 10 . We estimate [jnj that the limit published 
by the UAl collaboration in Rcf. should be shifted 
upward by about a factor of four. 

In conclusion, we have conducted a search for the 
FCNC decays b -> X s fi+fj,~ and B° -> /Jt+fj,-. We find 
no evidence for either decay. For the semi-inclusive de- 
cay b — > X s fi + n~ we set a 90% confidence level limit of 
B(b — > X s n + fj,~) < 3.2 x 10~ 4 . In view of our obser- 
vations we conclude ]l8| that this limit is more stringent 
than the best published limit on this decay rate. The SM 
prediction is 6xl0 -6 . 
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